{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run this, press \"*Runtime*\" and press \"*Run all*\" on your A100 Google Colab Pro instance!\n",
    "<div class=\"align-center\">\n",
    "<a href=\"https://unsloth.ai/\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\" width=\"115\"></a>\n",
    "<a href=\"https://discord.gg/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Discord button.png\" width=\"145\"></a>\n",
    "<a href=\"https://docs.unsloth.ai/\"><img src=\"https://github.com/unslothai/unsloth/blob/main/images/documentation%20green%20button.png?raw=true\" width=\"125\"></a></a> Join Discord if you need help + \u2b50 <i>Star us on <a href=\"https://github.com/unslothai/unsloth\">Github</a> </i> \u2b50\n",
    "</div>\n",
    "\n",
    "To install Unsloth your local device, follow [our guide](https://docs.unsloth.ai/get-started/install-and-update). This notebook is licensed [LGPL-3.0](https://github.com/unslothai/notebooks?tab=LGPL-3.0-1-ov-file#readme).\n",
    "\n",
    "You will learn how to do [data prep](#Data), how to [train](#Train), how to [run the model](#Inference), & [how to save it](#Save)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### News"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Introducing FP8 precision training for faster RL inference. [Read Blog](https://docs.unsloth.ai/new/fp8-reinforcement-learning).\n",
    "\n",
    "Unsloth's [Docker image](https://hub.docker.com/r/unsloth/unsloth) is here! Start training with no setup & environment issues. [Read our Guide](https://docs.unsloth.ai/new/how-to-train-llms-with-unsloth-and-docker).\n",
    "\n",
    "[gpt-oss RL](https://docs.unsloth.ai/new/gpt-oss-reinforcement-learning) is now supported with the fastest inference & lowest VRAM. Try our [new notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) which creates kernels!\n",
    "\n",
    "Introducing [Vision](https://docs.unsloth.ai/new/vision-reinforcement-learning-vlm-rl) and [Standby](https://docs.unsloth.ai/basics/memory-efficient-rl) for RL! Train Qwen, Gemma etc. VLMs with GSPO - even faster with less VRAM.\n",
    "\n",
    "Visit our docs for all our [model uploads](https://docs.unsloth.ai/get-started/all-our-models) and [notebooks](https://docs.unsloth.ai/get-started/unsloth-notebooks).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "%%capture\nimport os, re\nif \"COLAB_\" not in \"\".join(os.environ.keys()):\n    !pip install unsloth\nelse:\n    # Do this only in Colab notebooks! Otherwise use pip install unsloth\n    import torch; v = re.match(r\"[0-9]{1,}\\.[0-9]{1,}\", str(torch.__version__)).group(0)\n    xformers = \"xformers==\" + (\"0.0.33.post1\" if v==\"2.9\" else \"0.0.32.post2\" if v==\"2.8\" else \"0.0.29.post3\")\n    !pip install --no-deps bitsandbytes accelerate {xformers} peft trl triton cut_cross_entropy unsloth_zoo\n    !pip install sentencepiece protobuf \"datasets==4.3.0\" \"huggingface_hub>=0.34.0\" hf_transfer\n    !pip install --no-deps unsloth\n!pip install transformers==4.56.2\n!pip install --no-deps trl==0.22.2"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TGMWlrRdzwgf"
   },
   "source": [
    "### Unsloth\n",
    "\n",
    "`FastModel` supports loading nearly any model now! This includes Vision and Text models!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 188,
     "referenced_widgets": [
      "4b2d73a3caf34ebcb72eec59fbc3192f",
      "8d33b939153e424f8bcb6b9e20388ef8",
      "67498978d8194150af005c5f494ef30c",
      "3232fba0dcd647de94895c5ebfb667a0",
      "04809efd3db146c2aafd308f10200464",
      "d2af8a2712bb42ab860ebda80da376bc",
      "8524552679b94102b623aa616bbb9a02",
      "12468b27a098420796a040e86c01ed40",
      "a254522d6e95440ea53f217785d98a4d",
      "3ce1a50e1d624f4d9e679f148b4b31c7",
      "ecd5db31b0604c09b7a86c5d578b3580"
     ]
    },
    "id": "-Xbb0cuLzwgf",
    "outputId": "154e4c42-7998-4c49-9f5b-b443dc49e59d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\ud83e\udda5 Unsloth: Will patch your computer to enable 2x faster free finetuning.\n",
      "\ud83e\udda5 Unsloth Zoo will now patch everything to make training faster!\n",
      "==((====))==  Unsloth 2025.9.1: Fast Gemma3 patching. Transformers: 4.55.4.\n",
      "   \\\\   /|    NVIDIA A100-SXM4-40GB. Num GPUs = 1. Max memory: 39.557 GB. Platform: Linux.\n",
      "O^O/ \\_/ \\    Torch: 2.8.0+cu126. CUDA: 8.0. CUDA Toolkit: 12.6. Triton: 3.4.0\n",
      "\\        /    Bfloat16 = TRUE. FA [Xformers = 0.0.32.post2. FA2 = False]\n",
      " \"-____-\"     Free license: http://github.com/unslothai/unsloth\n",
      "Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b2d73a3caf34ebcb72eec59fbc3192f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from unsloth import FastModel\n",
    "import torch\n",
    "\n",
    "fourbit_models = [\n",
    "    # 4bit dynamic quants for superior accuracy and low memory use\n",
    "    \"unsloth/gemma-3-1b-it-unsloth-bnb-4bit\",\n",
    "    \"unsloth/gemma-3-4b-it-unsloth-bnb-4bit\",\n",
    "    \"unsloth/gemma-3-12b-it-unsloth-bnb-4bit\",\n",
    "    \"unsloth/gemma-3-27b-it-unsloth-bnb-4bit\",\n",
    "\n",
    "    # Other popular models!\n",
    "    \"unsloth/Llama-3.1-8B\",\n",
    "    \"unsloth/Llama-3.2-3B\",\n",
    "    \"unsloth/Llama-3.3-70B\",\n",
    "    \"unsloth/mistral-7b-instruct-v0.3\",\n",
    "    \"unsloth/Phi-4\",\n",
    "] # More models at https://huggingface.co/unsloth\n",
    "\n",
    "model, tokenizer = FastModel.from_pretrained(\n",
    "    model_name = \"unsloth/gemma-3-27b-it\",\n",
    "    max_seq_length = 2048, # Choose any for long context!\n",
    "    load_in_4bit = True,  # 4 bit quantization to reduce memory\n",
    "    load_in_8bit = False, # [NEW!] A bit more accurate, uses 2x memory\n",
    "    full_finetuning = False, # [NEW!] We have full finetuning now!\n",
    "    # token = \"hf_...\", # use one if using gated models\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SXd9bTZd1aaL"
   },
   "source": [
    "We now add LoRA adapters so we only need to update a small amount of parameters!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6bZsfBuZDeCL",
    "outputId": "ed01ad9a-aebd-48c0-e340-5609ca14ea7d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unsloth: Making `model.base_model.model.model.language_model` require gradients\n"
     ]
    }
   ],
   "source": [
    "model = FastModel.get_peft_model(\n",
    "    model,\n",
    "    finetune_vision_layers     = False, # Turn off for just text!\n",
    "    finetune_language_layers   = True,  # Should leave on!\n",
    "    finetune_attention_modules = True,  # Attention good for GRPO\n",
    "    finetune_mlp_modules       = True,  # SHould leave on always!\n",
    "\n",
    "    r = 8,           # Larger = higher accuracy, but might overfit\n",
    "    lora_alpha = 8,  # Recommended alpha == r at least\n",
    "    lora_dropout = 0,\n",
    "    bias = \"none\",\n",
    "    random_state = 3407,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vITh0KVJ10qX"
   },
   "source": [
    "<a name=\"Data\"></a>\n",
    "### Data Prep\n",
    "We now use the `Gemma-3` format for conversation style finetunes. We use [Maxime Labonne's FineTome-100k](https://huggingface.co/datasets/mlabonne/FineTome-100k) dataset in ShareGPT style. Gemma-3 renders multi turn conversations like below:\n",
    "\n",
    "```\n",
    "<bos><start_of_turn>user\n",
    "Hello!<end_of_turn>\n",
    "<start_of_turn>model\n",
    "Hey there!<end_of_turn>\n",
    "```\n",
    "\n",
    "We use our `get_chat_template` function to get the correct chat template. We support `zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, phi3, llama3, phi4, qwen2.5, gemma3` and more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "LjY75GoYUCB8"
   },
   "outputs": [],
   "source": [
    "from unsloth.chat_templates import get_chat_template\n",
    "tokenizer = get_chat_template(\n",
    "    tokenizer,\n",
    "    chat_template = \"gemma-3\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "Mkq4RvEq7FQr"
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "dataset = load_dataset(\"mlabonne/FineTome-100k\", split = \"train\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K9CBpiISFa6C"
   },
   "source": [
    "We now use `standardize_data_formats` to try converting datasets to the correct format for finetuning purposes!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "reoBXmAn7HlN"
   },
   "outputs": [],
   "source": [
    "from unsloth.chat_templates import standardize_data_formats\n",
    "dataset = standardize_data_formats(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6i5Sx9In7vHi"
   },
   "source": [
    "Let's see how row 100 looks like!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "dzE1OEXi7s3P",
    "outputId": "ebf1799c-df3f-44bc-b6c3-37128e2164be"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'conversations': [{'content': 'What is the modulus operator in programming and how can I use it to calculate the modulus of two given numbers?',\n",
       "   'role': 'user'},\n",
       "  {'content': 'In programming, the modulus operator is represented by the \\'%\\' symbol. It calculates the remainder when one number is divided by another. To calculate the modulus of two given numbers, you can use the modulus operator in the following way:\\n\\n```python\\n# Calculate the modulus\\nModulus = a % b\\n\\nprint(\"Modulus of the given numbers is: \", Modulus)\\n```\\n\\nIn this code snippet, the variables \\'a\\' and \\'b\\' represent the two given numbers for which you want to calculate the modulus. By using the modulus operator \\'%\\', we calculate the remainder when \\'a\\' is divided by \\'b\\'. The result is then stored in the variable \\'Modulus\\'. Finally, the modulus value is printed using the \\'print\\' statement.\\n\\nFor example, if \\'a\\' is 10 and \\'b\\' is 4, the modulus calculation would be 10 % 4, which equals 2. Therefore, the output of the above code would be:\\n\\n```\\nModulus of the given numbers is: 2\\n```\\n\\nThis means that the modulus of 10 and 4 is 2.',\n",
       "   'role': 'assistant'}],\n",
       " 'source': 'infini-instruct-top-500k',\n",
       " 'score': 4.774171352386475}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8Xs0LXio7rfd"
   },
   "source": [
    "We now have to apply the chat template for `Gemma-3` onto the conversations, and save it to `text`. We remove the `<bos>` token using removeprefix(`'<bos>'`) since we're finetuning. The Processor will add this token before training and the model expects only one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "1ahE8Ys37JDJ"
   },
   "outputs": [],
   "source": [
    "def formatting_prompts_func(examples):\n",
    "   convos = examples[\"conversations\"]\n",
    "   texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False).removeprefix('<bos>') for convo in convos]\n",
    "   return { \"text\" : texts, }\n",
    "\n",
    "dataset = dataset.map(formatting_prompts_func, batched = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ndDUB23CGAC5"
   },
   "source": [
    "Let's see how the chat template did! Notice there is no `<bos>` token as the processor tokenizer will be adding one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 209
    },
    "id": "gGFzmplrEy9I",
    "outputId": "2c485a4b-30f3-467b-ab7b-a65752dfe099"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'<start_of_turn>user\\nWhat is the modulus operator in programming and how can I use it to calculate the modulus of two given numbers?<end_of_turn>\\n<start_of_turn>model\\nIn programming, the modulus operator is represented by the \\'%\\' symbol. It calculates the remainder when one number is divided by another. To calculate the modulus of two given numbers, you can use the modulus operator in the following way:\\n\\n```python\\n# Calculate the modulus\\nModulus = a % b\\n\\nprint(\"Modulus of the given numbers is: \", Modulus)\\n```\\n\\nIn this code snippet, the variables \\'a\\' and \\'b\\' represent the two given numbers for which you want to calculate the modulus. By using the modulus operator \\'%\\', we calculate the remainder when \\'a\\' is divided by \\'b\\'. The result is then stored in the variable \\'Modulus\\'. Finally, the modulus value is printed using the \\'print\\' statement.\\n\\nFor example, if \\'a\\' is 10 and \\'b\\' is 4, the modulus calculation would be 10 % 4, which equals 2. Therefore, the output of the above code would be:\\n\\n```\\nModulus of the given numbers is: 2\\n```\\n\\nThis means that the modulus of 10 and 4 is 2.<end_of_turn>\\n'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[100][\"text\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "idAEIeSQ3xdS"
   },
   "source": [
    "<a name=\"Train\"></a>\n",
    "### Train the model\n",
    "Now let's train our model. We do 60 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "95_Nn-89DhsL"
   },
   "outputs": [],
   "source": [
    "from trl import SFTTrainer, SFTConfig\n",
    "trainer = SFTTrainer(\n",
    "    model = model,\n",
    "    tokenizer = tokenizer,\n",
    "    train_dataset = dataset,\n",
    "    eval_dataset = None, # Can set up evaluation!\n",
    "    args = SFTConfig(\n",
    "        dataset_text_field = \"text\",\n",
    "        per_device_train_batch_size = 2,\n",
    "        gradient_accumulation_steps = 4, # Use GA to mimic batch size!\n",
    "        warmup_steps = 5,\n",
    "        # num_train_epochs = 1, # Set this for 1 full training run.\n",
    "        max_steps = 30,\n",
    "        learning_rate = 2e-4, # Reduce to 2e-5 for long training runs\n",
    "        logging_steps = 1,\n",
    "        optim = \"adamw_8bit\",\n",
    "        weight_decay = 0.001,\n",
    "        lr_scheduler_type = \"linear\",\n",
    "        seed = 3407,\n",
    "        report_to = \"none\", # Use TrackIO/WandB etc\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "C_sGp5XlG6dq"
   },
   "source": [
    "We also use Unsloth's `train_on_completions` method to only train on the assistant outputs and ignore the loss on the user's inputs. This helps increase accuracy of finetunes!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "juQiExuBG5Bt"
   },
   "outputs": [],
   "source": [
    "from unsloth.chat_templates import train_on_responses_only\n",
    "trainer = train_on_responses_only(\n",
    "    trainer,\n",
    "    instruction_part = \"<start_of_turn>user\\n\",\n",
    "    response_part = \"<start_of_turn>model\\n\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Dv1NBUozV78l"
   },
   "source": [
    "Let's verify masking the instruction part is done! Let's print the 100th row again.  Notice how the sample only has a single `<bos>` as expected!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 209
    },
    "id": "LtsMVtlkUhja",
    "outputId": "d6571616-f5e2-42d8-db0b-da30f32721d5"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'<bos><start_of_turn>user\\nWhat is the modulus operator in programming and how can I use it to calculate the modulus of two given numbers?<end_of_turn>\\n<start_of_turn>model\\nIn programming, the modulus operator is represented by the \\'%\\' symbol. It calculates the remainder when one number is divided by another. To calculate the modulus of two given numbers, you can use the modulus operator in the following way:\\n\\n```python\\n# Calculate the modulus\\nModulus = a % b\\n\\nprint(\"Modulus of the given numbers is: \", Modulus)\\n```\\n\\nIn this code snippet, the variables \\'a\\' and \\'b\\' represent the two given numbers for which you want to calculate the modulus. By using the modulus operator \\'%\\', we calculate the remainder when \\'a\\' is divided by \\'b\\'. The result is then stored in the variable \\'Modulus\\'. Finally, the modulus value is printed using the \\'print\\' statement.\\n\\nFor example, if \\'a\\' is 10 and \\'b\\' is 4, the modulus calculation would be 10 % 4, which equals 2. Therefore, the output of the above code would be:\\n\\n```\\nModulus of the given numbers is: 2\\n```\\n\\nThis means that the modulus of 10 and 4 is 2.<end_of_turn>\\n'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(trainer.train_dataset[100][\"input_ids\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4Kyjy__m9KY3"
   },
   "source": [
    "Now let's print the masked out example - you should see only the answer is present:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 209
    },
    "id": "_rD6fl8EUxnG",
    "outputId": "1a4b6e1e-5b0a-42e0-abff-6f5b9e6de29b"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'                               In programming, the modulus operator is represented by the \\'%\\' symbol. It calculates the remainder when one number is divided by another. To calculate the modulus of two given numbers, you can use the modulus operator in the following way:\\n\\n```python\\n# Calculate the modulus\\nModulus = a % b\\n\\nprint(\"Modulus of the given numbers is: \", Modulus)\\n```\\n\\nIn this code snippet, the variables \\'a\\' and \\'b\\' represent the two given numbers for which you want to calculate the modulus. By using the modulus operator \\'%\\', we calculate the remainder when \\'a\\' is divided by \\'b\\'. The result is then stored in the variable \\'Modulus\\'. Finally, the modulus value is printed using the \\'print\\' statement.\\n\\nFor example, if \\'a\\' is 10 and \\'b\\' is 4, the modulus calculation would be 10 % 4, which equals 2. Therefore, the output of the above code would be:\\n\\n```\\nModulus of the given numbers is: 2\\n```\\n\\nThis means that the modulus of 10 and 4 is 2.<end_of_turn>\\n'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode([tokenizer.pad_token_id if x == -100 else x for x in trainer.train_dataset[100][\"labels\"]]).replace(tokenizer.pad_token, \" \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2ejIt2xSNKKp",
    "outputId": "0f672e49-9bbb-4d8c-f42b-5ed4be256916"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPU = NVIDIA A100-SXM4-40GB. Max memory = 39.557 GB.\n",
      "19.074 GB of memory reserved.\n"
     ]
    }
   ],
   "source": [
    "# @title Show current memory stats\n",
    "gpu_stats = torch.cuda.get_device_properties(0)\n",
    "start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
    "max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\n",
    "print(f\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\")\n",
    "print(f\"{start_gpu_memory} GB of memory reserved.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CNP1Uidk9mrz"
   },
   "source": [
    "Let's train the model! To resume a training run, set `trainer.train(resume_from_checkpoint = True)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "yqxqAZ7KJ4oL",
    "outputId": "729526d6-8c84-4318-df98-573fe58c807c"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "==((====))==  Unsloth - 2x faster free finetuning | Num GPUs used = 1\n",
      "   \\\\   /|    Num examples = 100,000 | Num Epochs = 1 | Total steps = 30\n",
      "O^O/ \\_/ \\    Batch size per device = 2 | Gradient accumulation steps = 4\n",
      "\\        /    Data Parallel GPUs = 1 | Total batch size (2 x 4 x 1) = 8\n",
      " \"-____-\"     Trainable parameters = 56,758,272 of 27,489,164,912 (0.21% trained)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unsloth: Will smartly offload gradients to save VRAM!\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='30' max='30' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [30/30 04:26, Epoch 0/1]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1.065200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1.393600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.431700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.121300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.994200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.409900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.687700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.071800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.804700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.691600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.821300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.952400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.884500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.539100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>0.760800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>0.562600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>0.941900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>0.748300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>0.706300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>0.876800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21</td>\n",
       "      <td>0.803600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>22</td>\n",
       "      <td>0.677400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>23</td>\n",
       "      <td>0.915200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>24</td>\n",
       "      <td>0.798800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25</td>\n",
       "      <td>0.538200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26</td>\n",
       "      <td>0.685500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27</td>\n",
       "      <td>0.710900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>28</td>\n",
       "      <td>0.709100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29</td>\n",
       "      <td>0.874500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30</td>\n",
       "      <td>0.903900</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trainer_stats = trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pCqnaKmlO1U9",
    "outputId": "02895ccb-a505-42b8-81f3-ceb4afd5ca9f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "291.7887 seconds used for training.\n",
      "4.86 minutes used for training.\n",
      "Peak reserved memory = 25.791 GB.\n",
      "Peak reserved memory for training = 6.717 GB.\n",
      "Peak reserved memory % of max memory = 65.2 %.\n",
      "Peak reserved memory for training % of max memory = 16.981 %.\n"
     ]
    }
   ],
   "source": [
    "# @title Show final memory and time stats\n",
    "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
    "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n",
    "used_percentage = round(used_memory / max_memory * 100, 3)\n",
    "lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\n",
    "print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n",
    "print(\n",
    "    f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\"\n",
    ")\n",
    "print(f\"Peak reserved memory = {used_memory} GB.\")\n",
    "print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n",
    "print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n",
    "print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ekOmTR1hSNcr"
   },
   "source": [
    "<a name=\"Inference\"></a>\n",
    "### Inference\n",
    "Let's run the model via Unsloth native inference! According to the `Gemma-3` team, the recommended settings for inference are `temperature = 1.0, top_p = 0.95, top_k = 64`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kR3gIAX-SM2q",
    "outputId": "f701cabd-c76f-4dc1-8d2f-374bc7ed2e7c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<bos><start_of_turn>user\\nContinue the sequence: 1, 1, 2, 3, 5, 8,<end_of_turn>\\n<start_of_turn>model\\nThe sequence follows the rule of adding the previous two numbers. \\n1+1 = 2\\n1+2 = 3\\n2+3 = 5\\n3+5 = 8\\n5+8 = 13\\n8+13 = 21\\nThe next two numbers in']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from unsloth.chat_templates import get_chat_template\n",
    "tokenizer = get_chat_template(\n",
    "    tokenizer,\n",
    "    chat_template = \"gemma-3\",\n",
    ")\n",
    "messages = [{\n",
    "    \"role\": \"user\",\n",
    "    \"content\": [{\n",
    "        \"type\" : \"text\",\n",
    "        \"text\" : \"Continue the sequence: 1, 1, 2, 3, 5, 8,\",\n",
    "    }]\n",
    "}]\n",
    "inputs = tokenizer.apply_chat_template(\n",
    "    messages,\n",
    "    add_generation_prompt = True, # Must add for generation\n",
    "    tokenize = True,\n",
    "    return_tensors = \"pt\",\n",
    "    return_dict = True,\n",
    ")\n",
    "outputs = model.generate(\n",
    "    **inputs.to(\"cuda\"),\n",
    "    max_new_tokens = 64, # Increase for longer outputs!\n",
    "    # Recommended Gemma-3 settings!\n",
    "    temperature = 1.0, top_p = 0.95, top_k = 64,\n",
    ")\n",
    "tokenizer.batch_decode(outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CrSvZObor0lY"
   },
   "source": [
    " You can also use a `TextStreamer` for continuous inference - so you can see the generation token by token, instead of waiting the whole time!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "e2pEuRb1r2Vg",
    "outputId": "3d7a402d-65da-4f6a-80c6-622276f45ebd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The sky is blue because of a phenomenon called **Rayleigh scattering**. Here's an explanation:\n",
      "\n",
      "1. **Sunlight consists of all colors.** When sunlight passes through the atmosphere, it interacts with air molecules (mainly nitrogen and oxygen).\n",
      "2. **Shorter wavelengths scatter more.** Different colors of light\n"
     ]
    }
   ],
   "source": [
    "messages = [{\n",
    "    \"role\": \"user\",\n",
    "    \"content\": [{\"type\" : \"text\", \"text\" : \"Why is the sky blue?\",}]\n",
    "}]\n",
    "inputs = tokenizer.apply_chat_template(\n",
    "    messages,\n",
    "    add_generation_prompt = True, # Must add for generation\n",
    "    tokenize = True,\n",
    "    return_tensors = \"pt\",\n",
    "    return_dict = True,\n",
    ")\n",
    "\n",
    "from transformers import TextStreamer\n",
    "_ = model.generate(\n",
    "    **inputs.to(\"cuda\"),\n",
    "    max_new_tokens = 64, # Increase for longer outputs!\n",
    "    # Recommended Gemma-3 settings!\n",
    "    temperature = 1.0, top_p = 0.95, top_k = 64,\n",
    "    streamer = TextStreamer(tokenizer, skip_prompt = True),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uMuVrWbjAzhc"
   },
   "source": [
    "<a name=\"Save\"></a>\n",
    "### Saving, loading finetuned models\n",
    "To save the final model as LoRA adapters, either use Huggingface's `push_to_hub` for an online save or `save_pretrained` for a local save.\n",
    "\n",
    "**[NOTE]** This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "upcOlWe7A1vc",
    "outputId": "72819696-3ad1-4c05-d088-a6f45cf92265"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['gemma-3/processor_config.json']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.save_pretrained(\"gemma-3\")  # Local saving\n",
    "tokenizer.save_pretrained(\"gemma-3\")\n",
    "# model.push_to_hub(\"HF_ACCOUNT/gemma-3\", token = \"...\") # Online saving\n",
    "# tokenizer.push_to_hub(\"HF_ACCOUNT/gemma-3\", token = \"...\") # Online saving"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AEEcJ4qfC7Lp"
   },
   "source": [
    "Now if you want to load the LoRA adapters we just saved for inference, set `False` to `True`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "MKX_XKs_BNZR",
    "outputId": "a5a3c8b5-ca2f-4cf4-f4b9-7d2a81f85329"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gemma is a family of open-weight models developed by Google DeepMind. It's designed to assist with a variety of tasks such as text generation, translation, and coding. There are two versions: Gemma 2B and Gemma 7B, which represent models with 2 billion and 7 billion parameters\n"
     ]
    }
   ],
   "source": [
    "if False:\n",
    "    from unsloth import FastModel\n",
    "    model, tokenizer = FastModel.from_pretrained(\n",
    "        model_name = \"gemma-3\", # YOUR MODEL YOU USED FOR TRAINING\n",
    "        max_seq_length = 2048,\n",
    "        load_in_4bit = True,\n",
    "    )\n",
    "\n",
    "messages = [{\n",
    "    \"role\": \"user\",\n",
    "    \"content\": [{\"type\" : \"text\", \"text\" : \"What is Gemma-3?\",}]\n",
    "}]\n",
    "inputs = tokenizer.apply_chat_template(\n",
    "    messages,\n",
    "    add_generation_prompt = True, # Must add for generation\n",
    "    tokenize = True,\n",
    "    return_tensors = \"pt\",\n",
    "    return_dict = True,\n",
    ")\n",
    "\n",
    "from transformers import TextStreamer\n",
    "_ = model.generate(\n",
    "    **inputs.to(\"cuda\"),\n",
    "    max_new_tokens = 64, # Increase for longer outputs!\n",
    "    # Recommended Gemma-3 settings!\n",
    "    temperature = 1.0, top_p = 0.95, top_k = 64,\n",
    "    streamer = TextStreamer(tokenizer, skip_prompt = True),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f422JgM9sdVT"
   },
   "source": [
    "### Saving to float16 for VLLM\n",
    "\n",
    "We also support saving to `float16` directly for deployment! We save it in the folder `gemma-3-finetune`. Set `if False` to `if True` to let it run!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "iHjt_SMYsd3P"
   },
   "outputs": [],
   "source": [
    "if False: # Change to True to save finetune!\n",
    "    model.save_pretrained_merged(\"gemma-3-finetune\", tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "z6O48DbNIAr0"
   },
   "source": [
    "If you want to upload / push to your Hugging Face account, set `if False` to `if True` and add your Hugging Face token and upload location!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "ZV-CiKPrIFG0"
   },
   "outputs": [],
   "source": [
    "if False: # Change to True to upload finetune\n",
    "    model.push_to_hub_merged(\n",
    "        \"HF_ACCOUNT/gemma-3-finetune\", tokenizer,\n",
    "        token = \"hf_...\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TCv4vXHd61i7"
   },
   "source": [
    "### GGUF / llama.cpp Conversion\n",
    "To save to `GGUF` / `llama.cpp`, we support it natively now for all models! For now, you can convert easily to `Q8_0, F16 or BF16` precision. `Q4_K_M` for 4bit will come later!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FqfebeAdT073"
   },
   "outputs": [],
   "source": [
    "if False: # Change to True to save to GGUF\n",
    "    model.save_pretrained_gguf(\n",
    "        \"gemma-3-finetune\",\n",
    "        tokenizer,\n",
    "        quantization_method = \"Q8_0\", # For now only Q8_0, BF16, F16 supported\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q974YEVPI7JS"
   },
   "source": [
    "Likewise, if you want to instead push to GGUF to your Hugging Face account, set `if False` to `if True` and add your Hugging Face token and upload location!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZgcJIhJ0I_es"
   },
   "outputs": [],
   "source": [
    "if False: # Change to True to upload GGUF\n",
    "    model.push_to_hub_gguf(\n",
    "        \"HF_ACCOUNT/gemma-finetune-gguf\",\n",
    "        tokenizer,\n",
    "        quantization_method = \"Q8_0\", # Only Q8_0, BF16, F16 supported\n",
    "        token = \"hf_...\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we're done! If you have any questions on Unsloth, we have a [Discord](https://discord.gg/unsloth) channel! If you find any bugs or want to keep updated with the latest LLM stuff, or need help, join projects etc, feel free to join our Discord!\n",
    "\n",
    "Some other links:\n",
    "1. Train your own reasoning model - Llama GRPO notebook [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb)\n",
    "2. Saving finetunes to Ollama. [Free notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb)\n",
    "3. Llama 3.2 Vision finetuning - Radiography use case. [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb)\n",
    "6. See notebooks for DPO, ORPO, Continued pretraining, conversational finetuning and more on our [documentation](https://docs.unsloth.ai/get-started/unsloth-notebooks)!\n",
    "\n",
    "<div class=\"align-center\">\n",
    "  <a href=\"https://unsloth.ai\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\" width=\"115\"></a>\n",
    "  <a href=\"https://discord.gg/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Discord.png\" width=\"145\"></a>\n",
    "  <a href=\"https://docs.unsloth.ai/\"><img src=\"https://github.com/unslothai/unsloth/blob/main/images/documentation%20green%20button.png?raw=true\" width=\"125\"></a>\n",
    "\n",
    "  Join Discord if you need help + \u2b50\ufe0f <i>Star us on <a href=\"https://github.com/unslothai/unsloth\">Github</a> </i> \u2b50\ufe0f\n",
    "\n",
    "  This notebook and all Unsloth notebooks are licensed [LGPL-3.0](https://github.com/unslothai/notebooks?tab=LGPL-3.0-1-ov-file#readme)\n",
    "</div>\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "A100",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "04809efd3db146c2aafd308f10200464": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "12468b27a098420796a040e86c01ed40": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3232fba0dcd647de94895c5ebfb667a0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3ce1a50e1d624f4d9e679f148b4b31c7",
      "placeholder": "\u200b",
      "style": "IPY_MODEL_ecd5db31b0604c09b7a86c5d578b3580",
      "value": "\u20075/5\u2007[00:07&lt;00:00,\u2007\u20071.17s/it]"
     }
    },
    "3ce1a50e1d624f4d9e679f148b4b31c7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4b2d73a3caf34ebcb72eec59fbc3192f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_8d33b939153e424f8bcb6b9e20388ef8",
       "IPY_MODEL_67498978d8194150af005c5f494ef30c",
       "IPY_MODEL_3232fba0dcd647de94895c5ebfb667a0"
      ],
      "layout": "IPY_MODEL_04809efd3db146c2aafd308f10200464"
     }
    },
    "67498978d8194150af005c5f494ef30c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_12468b27a098420796a040e86c01ed40",
      "max": 5,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_a254522d6e95440ea53f217785d98a4d",
      "value": 5
     }
    },
    "8524552679b94102b623aa616bbb9a02": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8d33b939153e424f8bcb6b9e20388ef8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d2af8a2712bb42ab860ebda80da376bc",
      "placeholder": "\u200b",
      "style": "IPY_MODEL_8524552679b94102b623aa616bbb9a02",
      "value": "Loading\u2007checkpoint\u2007shards:\u2007100%"
     }
    },
    "a254522d6e95440ea53f217785d98a4d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "d2af8a2712bb42ab860ebda80da376bc": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ecd5db31b0604c09b7a86c5d578b3580": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "state": {}
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}